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Abstract —This article addresses improvements on the design 
of the adaptive normalized matched filter (ANMF) for radar 
detection. It is well-acknowledged that the estimation of the 
noise-clutter covariance matrix is a fundamental step in adaptive 
radar detection. In this paper, we consider regularized estimation 
methods which force by construction the eigenvalues of the 
scatter estimates to be greater than a positive regularization 
parameter p. This makes them more suitable for high dimensional 
problems with a limited number of secondary data samples than 
traditional sample covariance estimates. While an increase of 
p seems to improve the conditioning of the estimate, it might 
however cause it to significantly deviate from the true covariance 
matrix. The setting of the optimal regularization parameter is a 
difficult question for which no convincing answers have thus far 
been provided. This constitutes the major motivation behind our 
work. More specifically, we consider the design of the ANMF 
detector for two kinds of regularized estimators, namely the 
regularized sample covariance matrix (RSCM), appropriate when 
the clutter follows a Gaussian distribution and the regularized 
Tyler estimator (RTE) for non-Gaussian spherically invariant 
distributed clutters. The rationale behind this choice is that the 
RTE is efficient in mitigating the degradation caused by the 
presence of impulsive noises while inducing little loss when the 
noise is Gaussian. 

Based on recent random matrix theory results studying the 
asymptotic fluctuations of the statistics of the ANMF detector 
when the number of samples and their dimension grow together 
to infinity, we propose a design for the regularization parameter 
that maximizes the detection probability under constant false 
alarm rates. Simulation results which support the efficiency of 
the proposed method are provided in order to illustrate the gain 
of the proposed optimal design over conventional settings of the 
regularization parameter. 

Index Terms —Regularized Tyler’s estimator, Adaptive Normal¬ 
ized Mached Filter, robust detection, Random Matrix Theory, 
Optimal design. 

I. Introduction 

The estimation of scatter matrices is of fundamental im¬ 
portance for space-time adaptive processing (STAP) which 
underlies the design of radar systems Cl, CD- In radar detec¬ 
tion for instance, it is well-acknowledged that a sufficiently 
accurate scatter matrix is key to enhancing the detection 
performance (see 0, [|4j] and references therein). In order to 
support a possible deficiency in samples (number of samples 
less than their dimensions), regularized covariance matrix 
estimation methods have been proposed 151. One regularized 
estimation method is given by the use of the regularized 
sample covariance matrix (RSCM). The RSCM fundamentally 
originates from the diagonal loading approach which can be 
traced back to the works of Abramovich and Carlson a, 
a. As a derivative of the sample covariance matrix (SCM), 
the RSCM inherits its main major limitation of exhibiting 
poor performances when observations contain outliers. This 


latter scenario is often modeled by assuming that observations 
are drawn from complex elliptical symmetric distributions 
(CES), originally introduced by Kelker 0. The inherent 
nature of these distributions to produce atypical observations 
makes the task of estimating the covariance matrix much 
more challenging. To tackle this issue, a class of covariance 
estimators coined robust estimators of scatter matrices have 
been proposed by Huber, Hampel, and Maronna 0-Hoi, 
and extended more recently by Ollila to the complex case 
0, 0, ED- Similar to the Gaussian case, the regularization 
technique has been applied to the robust Tyler estimator 02). 
yielding the so-called regularized Tyler estimator (RTE). While 
conventional robust covariance methods are undefined for too 
few samples (number of samples less than their dimensions), 
the existence of the RTE as well as the convergence of the 
associated recursive algorithm are two major findings which 
have recently been established in several works m-m. 
Unlike the RSCM, the RTE, as a derivative of the robust 
Tyler’s estimator, is resilient to the presence of outliers, 
thereby making it more suitable to radar applications, for 
which experimental evidence rules out Gaussian models for 
the clutter ED-ED. 

As far as regularized estimation methods are concerned, it is 
essential to determine a clever way of setting the regularization 
parameter. This question has essentially been investigated in 
ED, E2 for the RSCM and in Q3), ED for the RTE. 
Although yielding different expressions, these works have the 
common denominator of being merely based on a distance 
minimization between the RTE or the SCM and the true 
covariance matrix. It is thus not clear whether these choices 
will allow for good performances when applied to detection 
problems. We consider in this work the design of the adaptive 
normalized matched filter (ANMF) for radar detection. Firstly 
introduced by ll24l and analyzed in ES-ED, this scheme 
was shown to enjoy the interesting features of constant false 
alarm property with respect to the clutter power and covariance 
matrix. This detector is obtained by replacing in the statistic of 
the normalized matched filter (NMF) the covariance matrix by 
a given estimate ESI , which is computed based on secondary 
data observations, i.e., n signal free independent and identi¬ 
cally distributed (i.i.d.) observations. Of interest in this work 
are the cases where the RSCM or the RTE are used in place 
of the unknown covariance matrix. We will consider first the 
scenario where the detector operates over Gaussian correlated 
clutters and thus uses the RSCM as a replacement for the 
unknown covariance matrix, a scheme which will be referred 
to as ANMF-RSCM. In order to come up with an appropriate 
design for the ANMF detector, it is essential to characterize 
the behaviour of its corresponding false alarm and detection 
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probabilities. 

Under the assumption of fixed dimensions, such a task 
seems to be out of reach. This has led us to consider in |28l 
the regime wherein the number of secondary data samples 
and their dimensions grow simultaneously to infinity, thereby 
allowing for the use of advanced tools from random matrix 
theory. The asymptotic false alarm probability for the ANMF- 
RTE was in particular derived in l28l (yet only as a mere 
application example of the main mathematical result.). In order 
to allow for an optimal choice of the regularization parame¬ 
ter, these results need to be augmented with an asymptotic 
analysis of the detection probability. This constitutes the main 
contribution of our work. In particular, we extend the results 
of [|28ll to the ANMF-RSCM detector by showing that its 
corresponding statistic flucutates as a Rayleigh distribution 
when no target is present and, additionally, establish that it 
behaves like a Rice distributed random variable otherwise. 
Based on the asymptotic characterization of these distributions, 
we propose an optimal setting of the regularization parameter 
that maximizes the asymptotic probability of detection for any 
given false alarm probability. 

In a second part, we consider the case where the clutter 
is drawn from heavy tailed distributions. It is thus natural to 
assume that the detector uses the RTE, since the RSCM is 
vulnerable to the presence of outliers and may provide poor 
performances. This scheme will be coined ANMF-RTE. By 
exploiting recent results on the asymptotic behavior of the 
RTE estimator |[28l . | [29| , we prove that, up to a certain change 
of variable, the ANMF-RTE is asymptotically equivalent to 
the ANMF-RSCM when operating over Gaussian clutters. 
This argues in favor of the role of the RTE to retrieve the 
Gaussian performances while operating over heavy distributed 
clutters. Finally, we prove through simulations the superiority 
of our design to some of the adhoc recent settings of the 
regularization parameter that have recently been proposed. 
The gain of our design likely owes to the high accuracy 
of the derived asymptotic results in predicting the detection 
probability of the ANMF schemes. 

The remainder of the paper is organized as follows. In the 
first section, we introduce the considered problem. Then, we 
propose an optimal design approach for the ANMF-RTE and 
the ANMF-RSCM. Finally, we illustrate using simulations the 
gain of the proposed design method over conventional settings 
of the regularization parameter. 

Notations: Throughout this paper, we depict vectors in 
lowercase boldface letters and matrices in uppercase boldface 
letters. The notation (.)* stands for the transpose conjugate 
while tr(.) and (.)“ 1 are the trace and inverse operators. The 
notation ||.|| stands for the Euclidean norm for vectors and for 
spectral norm for matrices. The arrow —4 designates almost 
sure convergence. The statement X = Y defines the new 
notation A' as being equal to Y. 

II. Problem Statement 

We consider the problem of detecting a complex signal 
vector p corrupted by an additive noise as: 

y = ap+x 


where y £ C v represents the vector received by an N- 
dimensional array of sensors, x stands for the noise clutter and 
a is a complex scalar modeling the unknown target amplitude. 
The signal detection problem is phrased as the following 
binary hypothesis test: 

Hi : y = ap+x 
H 0 : y = x. 

Several models for the clutter x have been proposed. Among 
them, we distinguish the class of CES random variates 
which encompass most of the commonly encountered random 
models, including the standard Gaussian distribution, the K- 
distribution, the Weibull distribution and many others 0. A 
CES distributed random variable is given by: 

x = v/rC^w 

where r is a positive scalar random variable called the texture , 
Cat is the covariance matri>{] and w is an A'-dimensional 
vector independent of r, zero-mean unitarily invariant with 
norm ||w|| = \/~N. The quantity C^w is referred to as 
speckle. The design of an appropriate statistic to the above 
hypothesis test depends on the amount of knowledge that is 
available to the detector. If the clutter is Gaussian with known 
covariance matrix C y while a is unknown, the Generalized 
Likelihood Ratio (GLRT) for the detection problem in 0 
results in the following test statistic: 

^ |y*c^ P | 

1 N = —i = 

yy*c^ 1 yp*c^ 1 p 

which corresponds to the square-root statistic of the ANMF 
detector. The statistic Tjv has been derived independently 
by several works, thereby leading the corresponding detector 
to have many alternative names: the constant false alarm 
(CFAR) matched subspace detector (MSD) ll30l . the nor¬ 
malized matched filter (NMF) l3ll . or the Linear Quadratic 
GLRT (LQ-GLRT) ll32ll . If the clutter is elliptically distributed, 
optimal detection procedures based on the GLRT principle 
lead to statistics that depend on the distribution of the texture 
t. Nevertheless, a complete knowledge of the statistics of the 
signal and noise cannot be acquired in practice. A reasonable 
hypothesis, largely used in radar detection, is to assume that 
only p is known while a and the statistics of the noise are ig¬ 
nored. To handle this case, the use of 7 A whose optimality has 
only been shown in the Gaussian setting has been advocated 
as a good detection technique. Such a choice has particularly 
been motivated by the result of l24l showing the asymptotic 
optimality of Tat, when N becomes increasingly large, under 
the setting of compound-Gaussian distributed clutters. From 
the expression of Tat, it can be seen that the detector is only 
required to know Cat up to a scale factor, which is much less 
restrictive than the requirement of optimal detection strategies. 

Since the covariance matrix Cat is unknown in practice, 
a popular approach consists in replacing in Tat the unknown 
covariance matrix Cn by an estimate built on signal free i.i.d. 

1 Note that when the second order statistics exist, the scatter matrix is equal 
to the covariance matrix (up to a constant). 
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observations xi, - ■■ , x n , termed secondary data. The result¬ 
ing detector is called the adaptive normalized matched filter 
(ANMF). Several concurrent estimators of Cat can be used. 
The most popular one is the traditional sample covariance 
matrix (SCM) given by: 

1 ” 

R\ = - v' x^* 

n 

i =1 

which corresponds to the Maximum-Likelihood estimator 
(MLE) if the clutter is Gaussian distributed. However, in some 
scenarios where the available number of observations n is 
of the same order or smaller than N, the SCM, being ill- 
conditioned, will not lead to accurate detection result^} A 
practical approach that has received considerable attention is 
to regularize the SCM, thereby yielding the regularized SCM 
(RSCM) given by: 

Rat(p) = (1 — p)Rjv+pIiv, (2) 


where the parameter p £ [0,1] serves to give more or less 
importance to the sample covariance matrix Ry depending 
on the available number of samples. The ANMF that uses the 
RSCM as a plug-in estimator of C y will be referred to as 
ANMF-RSCM. 


While this regularization artifice has revealed efficient in 
handling the scarcity of the available samples, it has the serious 
drawback of fundamentally relying on the SCM. In effect, the 
SCM, even though suitable for Gaussian settings, is known to 
be vulnerable to outliers and thus leads to highly inefficient 
estimators when the samples are drawn from heavy tailed non- 
Gaussian distributions. A standard alternative to conventional 
sample covariance estimates is constituted by the class of 
robust-scatter estimators, known for their resilience to atypical 
observations. The robust estimator that will be considered in 
this work was defined in lfl4l as the unique solution C n{p) 
to: 


Cjv(p) = (1-p)-^ 
n 


X ? ;XJ 


^ ^ x *C Jv 1 (p)x i 


+ P^N- 


(3) 


t=1 


with p £ (max (0,1— , l]. This estimator corresponds 

to a hybrid robust-shrinkage estimator reminding Tyler’s M- 
estimator of scale Q2 and Ledoit-Wolf’s shrinkage estimator 
ED- We will thus refer to it as the Regularized-Tyler Estimator 
(RTE). Besides its robustness, the RTE has many interesting 
features. First, it is well-suited to situations where cn — — is 
large while standard robust scatter estimates are ill-conditioned 
or even undefined if N > n. By varying the regularization 
parameter p, one can move from the unbiased Tyler-estimator 
|[34l (p = 0) to the identity matrix (p = 1) which represents 
a crude guess for the unknown covariance Cjy. Its relation 
to the Tyler’s estimator has recently been reported in [15j by 
viewing it as the solution of a penalized M-estimation cost 
function. We will denote by ANMF-RTE the ANMF detector 
that uses the RTE instead of the unknown covariance matrix. 

Upon replacing in Tat the unknown covariance matrix by a 
regularized estimate, be it the SCM or the RTE, the question 


- Traditionally, it is assumed that 2TV observations are required to ensure 
good performances of the sub-optimal filtering, i.e., a 3 dB loss of the output 
SNR compared to optimal filtering (23. 


of how should the regularization parameter p be set naturally 
arises. Recent previous works dealing with this issue propose 
to set p in such a way as to minimize a certain mean-squared- 
error between Cat and Cai [?], G3- While easy-to-compute 
estimates of these values of p were provided, one of the 
major criticism to these choices is that they are performed 
regardless of the application under consideration. In particular, 
a more relevant choice to the application under study consists 
in selecting the values of p that maximize the probability of 
detection while keeping fixed the false alarm probabilities. 
These values will be considered as optimal in regards of radar 
detection applications. 

To this end, one needs to characterize the distribution of 
yRSCM(^) anc j J-RTE(p) gi ven by; 


T* SCM (p) 


/f^RTE 
1 N 


(p) 


y *Rjv 1 (p)p 

V^Rw^yvVRjv^p 

y*c^(p)p 

\J y* c n 1 {p)y\J p* c^ 1 ( P ) p 


(4) 

(5) 


under hypotheses H 0 and Hi. For fixed N and n, this is not an 
easy task and in our opinion would not lead, if ever feasible, 
to easy-to-compute expressions for the optimal values of p. In 
this paper, we relax this restrictive assumption by considering 
the case where N and n go to infinity with —> c £ (0, oo). 
This in particular enables leveraging the recent results of |[28l 
that will be reviewed in Section IIV-AI 


III. Optimal design of the ANMF-RSCM detector: 

Gaussian clutter case 

In this section, we consider the case of a Gaussian clutter. In 
other words, we assume that all the secondary data xi, ■ ■ ■ , x„ 
are drawn from Gaussian distribution with zero-mean and co- 
variance Cat. For p £ (0,1], we define the RSCM as in (0 and 
corresponding statistic Tff SCM . In order to pave the way to¬ 
wards an optimal setting of the regularization coefficient p, we 
need to characterize the asymptotic false alarm and detection 


probabilities under the assumptions that Cn = 


A N 


c. That is, 


provided Hq or Hi is the actual scenario, (y = x or y = ap + 


x), we shall evaluate the probabilities 


t r scm > r| Ho 


and 


fRSCM > p| Fl 


for T > 0. Before going further, 
we need to stress that some extra assumptions on the order 
of magnitude of a and T with respect to N should be made 
to avoid getting trivial results. Indeed, it appears that under 
H 0 , the random quantities -l=y*R- 1 (p)^, j f y*R~ 1 (p)y, 

and p*R /v 1 (p) are standard objects in random matrix 
theory, which converge almost surely to their means when 
both N and n grow to infinity with the same pace lf35l . As 
a result, since -iy*Rj v 1 (p)p 0, Tj$ SGM —>■ 0 for 
all T > 0, which does not allow to infer much information 
about the false alarm probability. It turns out that the proper 
scaling of T should be T = N~?r for some fixed r > 0, 
an assumption already considered in ||28l . Similarly, one can 
see that under Hi, the presence of a signal component in 
y causes T^ SCM to converge almost surely to some positive 
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constant if a does not vary with N. Therefore, for T = N 2 r , 


T RSCM > P| ffl 


—> 1. In order to avoid this trivial 


statement, we shall assume that a = N~^a for some fixed 
a > 0 with ||p|| = N. In practice, this means that the 
dimension of the array is sufficiently large to enable working 
in low-SNR regimes. 

Prior to introducing the results about the false alarm and 
detection probabilities, we shall introduce the following as¬ 
sumptions and notations: 


Assumption A-l. For i £ {1, • • • , n }, x* = with: 

• wi,--- , w n are Nxl independent standard Gaussian 
random vectors with zero-mean and covariance In, 

• Cat £ C NxN is such that limsup ||Cjv|| < oo and 
77 tr Cat = 1 , 

• lirri infyv -^P*CatP > 0. 


Note that the normalization -T tr Cat = 1 is not a restricting 
constraint since the statistics under study are invariant to the 
scaling of Cat. The last item in Assumption Q] is required 
for technical purposes in order to ensure that the considered 
statistic exhibits fluctuations under Hq and H±. In practice, 
this assumption implies that the steering vector does not lie in 
the null space of the covariance matrix Cat. 

Denote for 2 £ C\K.+ by toat(z) the unique complex 
solution to: 


rriN(z) = (- z+cn(1~P) 

x T tr Cat (Ijv + (l-p)TOAr(^)CAr) _1 ^ 


-1 


where: 

_ 2 / ^ a 1 P*C N Q 2 N (p)p 

a N,SCM\P — 77 , 7 7 1 , r , 77 7 7 

2 P Qat(p)p^ tr C n Qn(p) 

1 

X l-c(l-p) 2 m N (-p) 2 ^ tr C 2 N Q, 2 N (p) 

and Qn{p) = (Itv + (1- p)m N {-p)C N )~ 1 . 

The uniformity over p of the convergence result in The¬ 
orem [I] is essential in the sequel. It obviously implies the 
pointwise convergence for each p > 0 but, more importantly, it 
will allow us to handle the convergence of the false alarm prob¬ 
ability when random values of the regularization parameter 
are considered. This feature becomes all the more interesting 
knowing that the detector is required to set the regularization 
parameter based on random received secondary data. Note that, 
for technical issues, a set of the form [0, k), where n > 0 is 
as small as desired but fixed, has to be discarded from the 
uniform convergence region. 

The result of Theorem Q] provides an analytical expression 
for the false alarm probability. Since this expression depends 
on the unknown covariance matrix, it is of practical interest 
to provide a consistent estimate for it: 

Proposition 2. For p £ (0,1), define 

i _ p*R y 2 (p)p 

_ P p»R^p _ 

l-CAT + ^trR^(p)) (l-£trR^(p 


&N,SCm(p) — 


that satisfies S(z)3(mAr(z)) > 0 or unique positive if 7 < 0. 
The existence and uniqueness of m.N{z) follows from standard 
results of random matrix theory ll36l . It is a deterministic 
quantity, which can be computed easily for each z using 
fixed-point iterations. In our case, it helps characterize the 
asymptotic behavior of the empirical spectral measure of the 
random matrix (l-p)T £)"=i x,;x*0 
Define also for n > 0, 7£® CM as: 

= [«, 1] • 


With these notations at hand, we are now ready to analyze 
the asymptotic behaviour of the false alarm and detection 
probabilities. The proof for the following Theorem will not 
be provided since, as we shall see in Section IIV1 it follows 
directly by applying the same approach used in |28l . 


Theorem 1 (False alarm probability). As N,n ^ oo with 

Cn —» c £ (0, oo). 


sup 

pe-RSCM 


T, 


RSCM / 


N {p) > 7n ih °. 


Ap) 


3 Let On = if X]£i be the empirical spectral measure of the 
random matrix —^ x i x * with Ai,--- ,Xn the eigenvalues of 

x ? ;x*. Denote by ttin(z) its Stieltjes transform given by 
th N {z) = f (t—z)- 1 P N (dt) = T Then ’ ( l uantit y m N(z) is 

the Stieljes transform of a certain deterministic measure hn, (i.e, tyin{z) = 
f (t — z)~ 1 jiiN(dt)) which approximates in the almost sure sense tHn^z) 
(i.e., mu (z) — rriN (z) -^4 0.). 


and let d 2 NSCM (l) = lim pn a 2 NSCM (p) = p * ^" p . Then, 
we have, for any k > 0, 


sup \cr%, scm(p)~ ,scm(p)| ~~0. 

pG^CM 


The proof of Proposition [2] follows along the same lines as 
that of Proposition 1 in Il28ll and is therefore omitted. 

We will now derive the asymptotic equivalent for 


T*scm ( p ) > 


\Hi 


_ l n \yj " viv 

tor y is supposed to be given by: 


, where under H i the received vec- 


H i : 


y = 


^=p+ 

Vn 


with x distributed as the Xj’s in Assumption!]] The following 
results constitute the major contribution of the present work. 
They will lead in conjunction with those of Theorem [8] and 
Proposition [2] to the optimal design of the ANMF-RSCM. 


Theorem 3 (Detection probability). As N, n —> oo with cn —> 
c, we have for any k > 0 


sup 

pgTjSCJM 


T« bOM (p) > -=|Ri 


s/N 


-Qi ( 5scm(p), 


cat,scm(p) 


0. 
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where Q\ is the Marcum Q-functiot 0 while cat,scm is given 
in Theorem □ and gsc&i( p) is given by: 


9scm{ p) = 


\Jl-c(l-p) 2 m(-p) 2 ± tr C 2 N Q 2 N (p) 


y/p*C N Q 2 N {p)p 


*\l ^a\p*Q N (p)p\. 


Proof: See Appendix lAl ■ 

According to Theorem Q] and Theorem [3] T^ SCM (p) be¬ 
haves differently depending on whether a signal is present 
or not. In particular, under H 0 , \/NT^ SCM (p) behaves 
like a Rayleigh distributed random variate with parameter 
<T/v,scm(p) while it becomes well-approximated under Hi by 
a Rice distributed random variable with parameters </scm(p) 
and <jn,scm(p)- It is worth noticing that in the theory of radar 
detection, getting a false alarm and a detection probability dis¬ 
tributed as Rayleigh and Rice random variables is among the 
simplest cases that one can ever encounter, holding only, to the 
best of the authors’ knoweldege, if white Gaussian noises are 
considered [?, p. 188]. We believe that the striking simplicity 
of the obtained results inheres in the double averaging effect 
that is a consequence of the considered asymptotic regime. 
This is to be compared to the quite intricate expressions for 
the false alarm probability obtained under the classical regime 
of n tending to infinity while N is fixed [j40| . 

We will now discuss the choice of the regularization param¬ 
eter p and the threshold r. In accordance with the theory of 
radar detection, we aim at setting p and r in such a way to keep 
the asymptotic false alarm probability equal to a fixed value 
ij while maximizing the asymptotic probability of detection. 
From Theorem Q] one can easily see that the values of r and 
p that provide an asymptotic false alarm probability equal to 
r/ should satisfy: 


r 

0 jv,scm(p) 


•\/—2 log 77 . 


From these choices, we have to take those values that max¬ 
imize the asymptotic detection which is given, according to 
Theorem [3] by: 


<9i ( PSCm(p),- 7 —r ) ■ 

V o'n,scm{p) / 

The second argument of Qi should be kept fixed in order 
to ensure the required asymptotic false alarm probability. As 
the Marcum-Q function increases with respect to the first 
argument, the optimization of the detection probability boils 
down to considering the following values of p: 


p £ argmax {/scm(p)} 


where: ^ 

/scm(p) = (p) 

However, the optimization of /scm(p) is not possible in prac¬ 
tice, since the expression of f scm(p ) features the covariance 


4 Qi(a,b) = f^~°°xex p ( - T ‘ f' j to( o,x)dx where Jo is the zero-th 
order modified Bessel function of the first kind. 


matrix C N which is unknown to the detector. Acquiring a 
consistent estimate of /scm(p) based on the available Rjv is 
thus mandatory. This is the goal of the following Proposition. 

Proposition 4. For p £ (0,1), define /scm( p) as: 

(p*Rw 1 (p)p) (1-p) (l-c-f-^ptrR^ 1 ^)) 
/scm(p) =- . - 

p Rjv (p)p-pp*Rw ( p ) p 

and let /scm(1) — lim P ti /scm(p) = —~—. Then, we have: 

P p 


sup 

PG-R.S CM 


/scm (p) —/scm (p) 


where we recall that TZ^. CM = [n, 1], 


Proof: See Appendix [B] ■ 

Since the results in Proposition [4] and Theorem [3 are 
uniform in p, we have the following corollary: 

Corollary 5. Let f scm(p ) be defined as in Proposition [3 
Define p* N as any value satisfying: 


p* N £ argmax 

pg-RSCM 



Then, for every r > 0, 

P ( '/NT n (p* n ) > r\H r) 


Proof: The proof is similar to that of Corollary 1 of ll28ll 

and is thus omitted. ■ 

From Corollary |5] the following design procedure leads to 
optimal performance detection results: 

• First, setting the regularization parameter to one of the 
values maximizing /scm(p): 

p* N £ argmax i /scm(p) \ (6) 

pg-rscm f J 

• Second, selecting the threshold r as: 

r = d N ,scM{p* N )\/-^ogV (7) 

IV. Optimal design of the ANMF-RTE: 

Non-Gaussian clutter 

This section discusses the design of the ANMF-RTE de¬ 
tector in the case where the clutter is non-Gaussian. In 
particular, we assume that the secondary observations satisfy 
the following assumptions: 

1 

Assumption A-2. For i £ {1, • ■ ■ , n}, Xj = y/riCfwi = 
y/rizi where 

• wj,-" , w n are N x 1 independent unitarly invariant 
complex zero-mean random vectors with ||wi|| 2 = N, 

• Cat £ C A, xA r is such that lim sup 1111 < 00 and 
jf tr Cat = 1. 

• n > 0 are independent of w,;. 

• liminf ■^p*Ca/'P > 0. 
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The random model described in Assumption[2]is that of CES 
distributions which encompass a wide range of observation 
distributions obtained for different settings of the statistics of 
Tj. Prior to stating our main findings, we shall first review 
some recent results concerning the asymptotic behaviour of 
the RTE in the asymptotic regime. 


A. Background 

This section reviews the recent results in j28l about the 
asymptotic behaviour of the RTE estimator. 

Recall that the RTE is defined, for p £ (max { 0,1 — } , l], 

as the unique solution to the following equation: 


C n(p) = (1 -p) 


sr 


n " 4x*C Ar 1 (p)x 1 


■ + pi-N- 


z—1 


The study of the asymptotic behaviour of robust-scatter esti¬ 
mators is much more challenging than that of the traditional 
sample covariance matrices. The main reasons are that, first, 
robust estimators of scatter do not have closed-form expres¬ 
sions and, second, the dependence between the outer-products 
involved in their expressions is non-linear, which does not 
allow for the use of standard random matrix analysis. In 
order to study this class of estimators, new technical tools 
based on different rewriting of the robust-scatter estimators 
have been developed by Couillet et al. Il29l . Il37ll . Il38l . The 
important advantage of these techniques is that they suggest to 
replace robust estimators by asymptotically equivalent random 
matrices for which many results from random matrix theory 
are applicable. In particular, the RTE estimator defined above 
has been studied in ll28l and has been shown to behave in the 
regime where N,n —> oo in such a way that cjv —> c £ (0, oo) 
similar to Sjy(p) given by: 


S n(p) = 


1 — 1 _ _ 

-y^ +P i N , (8) 


7 n(p) 1-{1-p)c N n^ 


where 7 jv(p) is the unique solution to: 

/ lN(p)p+{l-p)t UN ^' 

More specifically, the following theorem applies: 

Theorem 6 ( 11291 ). For any n > 0 small, define '/2.J < rE = 
[«:+max(0,1 —c _1 ), l]. Then, as N,n —»• oo with c/v —> c £ 
(0, oo), we have: 


sup 

pG'R)J TE 


C N (p)-S N (p) 


0. 


Theorem [6] establishes a convergence in the operator norm 
of the difference Cn{p) — Sn(p). This result allows one to 
transfer the asymptotic first order analysis of many functionals 
of Cn{p) to S n(p)- However, when it comes to the study 
of fluctuations, this result is of little help. Indeed, although 
Theorem [6] can be easily refined as 


sup 

p£R.K T ‘ 


N 2- 


C N (p)-S N (p) 


0. 


for each e > 0, the above convergence does not suffice 
to obtain the convergence of most of the commonly used 
functionals which involve fluctuations of order ~ or iV -1 
(e.g. quadratic forms of Cn{p) or linear statistics of the eigen¬ 
values of Cjv(p)). While a further refinement of the above 
convergence seems to be out of reach, it has recently been 
established in l28l that the fluctuations of special functionals 
can be proved to be much faster, mainly by virtue of an 
averaging effect which cancels out terms fluctuating at lower 
speed. In particular, bilinear forms of the type a*C^(p)b were 
studied in ll28l . where the following proposition was proved: 


Proposition 7. Let a, b £ C N with ||a|| = ||b|| = 1 
deterministic or random independent ofx i, • • • , x„. Then, as 
N, n —> oo, with cn — > c £ (0, oo), for any e > 0 and every 
k £ Z, 


sup iV 1 ” 6 

pG-R« TE 


a*C k N (p)b-a*S k N (p)b 


0. 


where 1Z ETE is defined as in Theorem® where k £ Z in any 
power of the matrices Cjv and Sjy. 


Some important consequences of Proposition [7] need to be 
stated. First, we shall recall that, while the crude study of the 
random variates a*C^(p)b seems to be intractable, quadratic 
forms of the type a*S^(p)b are well-understood objects 
whose behavior can be studied using standard tools from 
random matrix theory j39]. It is thus interesting to transfer 
the study of the fluctuations of a*C^(p)b to a*S^(p)b. 
Proposition [7] achieves this goal by taking e < i. Not only 
does it entail that a*C^(p)b fluctuates at the order of N~ 2 
(since so does a*S^(p)b) but also it allows one to prove that 
a*C^(p)b and a*S^(p)b exhibit asymptotically the same 
fluctuations. Similar to l28l . our concern will be rather focused 
on the case k = —1. In the next section, we will show how this 
result can be exploited in order to derive the receiver operating 
characteristic (ROC) of the ANMF-RTE detector. 


B. Optimal design of the ANMF-RTE detector 


As explained above, in order to allow for an optimal 
design of the ANMF-RTE detector, one needs to characterize 
the distribution of T^ TE (p) under hypotheses Hq and Hi. 
Using Proposition [7] we know that the statistic T ETE (p) 
which cannot be handled directly, has the same fluctuations 
as T* TE (p) obtained by replacing C n(p) by S n(p)- That is: 


T^rte 

1 n 


(P) = 


y*s jv (p)p 


P*S Ar (p)pJy*S N (p )y 


where Sjv(p) is given by ©. 

Let ^P = p ( p + 1n \ p ) i-(i~-P)c ) ■ Then ’ Siv(p) = 

PP~ 1 TLn(p), where, with a slight abuse of notation, we 
denote by R n(p) the matrix (1 —p)i ^” =1 z.jZ* +pljsr- Since 
T ETE (p) remains unchanged after scaling of S n(p) an d y, 
we also have: 


T^(P) = 


hy*R-N(p)p 




p*R JV 1 (p)p x / iy*R 7 V 1 (p)y 
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where t = 1 under Hq. It turns out that, conditionally to r, the 
fluctuations of the robust statistic T^ TE (p) under Hq or Hi 
are the same as those obtained in Theorem |Tj and Theorem 0 
once a is replaced by and p by /{§ As a consequence, we 
have the following results: 

Theorem 8 (False alarm probability, Il28l ). As TV, n —> oo 
with cat —»• c £ (0, oo). 


sup 

p^TZRTE 


rpRTE 
1 N 


(P) > yy I Ho 


— e 


zip) 


—y 0 , 


where p i—»• p is the aforementioned mapping and 


<j at,rte(p) — 


1 


p*CjvQ|rO)p 


2 P*Qat(p)p^ tr C n Qn(p) 


(l-c(l-p) 2 m(-p) 2 jftrC 2 N Q 2 N (p)) 

with Qat(p) = (ljv + (l — p)tn(— p)Cjy) 1 . 

Theorem 9 (Detection probability). As TV, n — > oo with c,y 

c £ (0, oo). 


sup 

pe^ TE 


2 * (p) > -=|ffr 




-E 


Q i ( Prte(p), 


<7 at,rte (p) 


0, 


where the expectation is taken over the distribution of r, 
<T/v,rte(p) has the same expression as in Tlieorem\S\and 


Srte(p)= 


y / l-c(l-p) 2 TO(-p)^trC^Q^(p) 


P*CjvQat(p)p 


sup 

pen* TE 


rnRT E 
1 N 


(p)> 7n lH " T . 


—Q i ( 9rte(p), 


0. 


ctn,rte(p) , 

The result thus follows by noticing the following inequality 


sup 

p^TZRTE 


rpRT E 
1 N 


(P) > ^IH. 


Q i ( 9rte{ p), 


-E 


< E sup 

pg7? RTE 


- Q i 9rte{ p), 


&n,rte{p) J _ 


T« te ( P )>—\H 1 ,t 


&n,rte(p) 


and resorting to the dominated convergence theorem. ■ 
Similar to the Gaussian case, we need to build consistent 
estimates for er^ jRTE (p) and /rte(p) given by: 

n~ 

/rte(p) = ^ 2 <?rte(p) 

A consistent estimate for RTE (p) was provided in (28l : 

Proposition 10 (Proposition 1 in 1281 ). For p £ 
(max({0,1 —c)^ 1 } , l). Define, 

p*C- 2 (p)p 


'TV, RTE 


(p) = o 


1 ~P 


P* C AT (P)P 


2 (1 —cjv+cjvp) (1 — p) 


and let cr)y RTE ( 1 ) = lim p -|-i d 2 N (p). Then, we have: 


sup |oat,rte(p)~ <5'jv j R XE (p)| 
pS7?, ETE 


0. 


Similar to the Gaussian clutter case, acquiring a consistent 
estimate for /rte(p) is mandatory for our design. We thus 
prove the following Proposition: 

Proposition 11. For p £ (max{0,1 — cfj 1 } , l), let 
/rte(p) = (p*C^(p)p) ^trCjv(p)-p^ 

(1 — cn+Cnp) 2 _ 

P*Cf f 1 {p)p-pp*Cf t 2 {p)p 

and /rte — lim pt i /rte(p)- Then, we have: 


sup 

pg7 jR T E 


x V Tv7 a l p *Q JV -) p l' 

and Qi is the Marcum Q-function. 

Proof: Since the fluctuations of the robust statistic 
TfP E {p) is the same as that of T^ SGM (p) when a is replaced 
by A=, we have for any fixed r. 


/rte (p) —/rte (p) —A 0 . 

Proof: The proof follows by first replacing R v ' (p) by 
R^y 1 (p) and p by p in the results of Proposition [4] and using 
the convergences 


sup 

P&R ETE 


sup 

P&n ETE 


Cn(p) 


ifrCjv(p) 


— Rjv(p) 


0. 


pCat(p)— pRw(p) 


0. 


and 


--T7 tr Civ(p) 
p TV 


0. 


5 Note that vector y can be assumed to be Gaussian without impacting the 
asymptotic distributions of s/NTP te under Hq and II \. 


Since the results in Proposition [TT] and Theorem [9] are 
uniform in p, we have the following corollary: 

Corollary 12. Let /rte(p) be defined as in Proposition [5] 
Define p* N as any value satisfying: 

p* N £ arg^nax E {/rte(p)} • 

Then, for every r > 0, 

P (VNT n (p* n ) > 




















































Using the same reasoning as the one followed in the Gaus¬ 
sian clutter case, we propose the following design strategy: 

• First, set the regularization parameter to one of the values 
maximizing f rte{p)- 

p* N G arg^nax E {f RTE (p)} 5 

• Second, set the threshold to r 

T = &N,RTE ( P*N ) V~ 2 l°g V 

where rj is the required false alarm probability. 

V. Numerical results 
A. Gaussian clutter 

In a first experiment, we consider the scenario where the 
clutter is Gaussian with covariance matrix Cj^ of Toeplitz 
form: 

[Civlij = | (pi-jy i>"j ’ l & l e ]o,i[, (9) 

where we set b = 0.96j N = 30 and n = 60. The steering 
vector p is given by 

p = a(6>) (10) 

where 9 > [a(0)] fe = e ~T n ' ksin ( e ) j n this experiment, 9 is 

set to 20°. For each Monte Carlo trial, the simulated data 
consists of y = ap+x and the secondary data yi,--- , y„ 
which are used to estimate p* N and to compute Rjv(p^f). In 
particular, the shrinkage parameter and the threshold value are 
determined using (|6| and 0. We have observed from the 
considered numerical results that fsCM(p) is unimodal and 
thus the maximum can be obtained using efficient line search 
methods. For comparison, we consider two other designs: the 
first one is based on the regularization parameter derived in 
the work of Chen et al. Il22l Equation (19)] (we denote by 
Pchen the corresponding coefficient) while the second one 
corresponds to the non-regularized ANMF (p = 0). . In order 
to satisfy the required false alarm probability, we assume for 
the first design that the threshold f c h e n is given by: f c hen = 
o'N,sCM{pchen)V~ 2log p. For the non-regularized ANMF to 
satisfy the false alarm probability, the threshold value is set 
based on Equation 11 in j27l . Figure Q] represents the ROC 
curves of both designs for different values of a = as^N, 
namely a = 0.1,0.25,0.5, along with that of the theoretical 
performances of our design. We note that for all SNR ranges, 
the proposed algorithm outperforms the design based on the 
regularization parameter p c hen and the gain becomes higher 
as a increases. It also outperforms the non-regularized ANMF 
detector. Moreover, the performances of the proposed design 
correspond with a good accuracy to what is expected by our 
theoretical results. 

In order to highlight the gain of the proposed design over 
the most interesting range of low false alarm probabilities, we 
represent in Fig. [2] the obtained ROC curves when a = 0.8 and 
the false alarm probability spanning the interval [0.001,0.05]. 


Proposed design 
Theory 

Design using p chen and r chen j22| 
Design without regularization p = 0 



Fig. 1. ROC curves of ANMF-RSCM designs for a = 0.1, 0.25, 0.5, p = 
a(0) with 9 = 20°, N = 30, n = 60: Gaussian setting 


— Proposed design 

Theory 

■■■' Design using p c he n and f che n J22] 
Design without regularization p = 0 



Fig. 2. ROC curves of ANMF-RSCM designs for a = 0.9, p = a (9) with 
9 = 20°, N = 30, n = 60: Gaussian setting 
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B. Non-Gaussian clutter 

In a second experiment, we proceed investigating the per¬ 
formance of the proposed design in the case where a non- 
Gaussian clutter is considered. In particular, we consider the 
case where the clutter is drawn from a /f-distribution with zero 
mean, covariance Cat, and shape v = 0.5 0. The covariance 
matrix has the same form as in ([9]) with b = 0.96j but with 
N = 30 and n = 60. 

Similar to the Gaussian clutter case, we consider for the 
sake of comparison the concurrent design based on the reg¬ 
ularization parameter derived in the work of Olilla and Tyler 
in 02 Equation (19)]. We denote by p 0 mia and f 0 mia the 
corresponding regularization coefficient and threshold. Note 
that, according to our theoretical analysis, the threshold f 0 nii a 
should be set to f 0 mia = <J/v, rte (/5 0 mia) y /~ 2 log V in order 
to satisfy the required false alarm probability. The results are 
depicted in Figure 0 We note that for all SNR ranges, the 
proposed method achieves a gain over the design based on 
the regularization coefficient proposed by Olilla et al. We also 
observe that, similar to the first experiment, the gain increases 
as a grows but with a lower slopqS 

In a last experiment, we investigate the impacts of a and 
the distribution shape v. Figure Q] represents the detection 
probability with respect to a when the false alarm probability 
is fixed to 0.05. We note that for small values of a, higher 
detection probabilities are achieved when the distribution of 
the clutter is heavy-tailed (small v), whereas the opposite 
occurs for large values of a. In order to explain this change 
in behavior, we must recall that heavy-tailed clutters (small 
v) are characterized by a higher number of occurrences of 
r in the vicinity of zero and at the same time more frequent 
realizations of large values of r. If a is small, the improvement 
in detection performances achieved by heavy-tailed clutters is 
attributed to the artificial increase in SNR over realizations of 
small values of r. As a increases, the power of the signal of 
interest is high enough so that the effect of realizations with 
large values of r becomes dominant. The latter, which are 
more frequent for small values of v , are characterized by high 
levels of noises, thereby entailing a degradation of detection 
performances. 


VI. Conclusion 

In this paper, we address the setting of the regularization 
parameter when the RSCM or the RTE are used in the ANMF 
detector statistic as a replacement of the unknown covari¬ 
ance matrix, thereby yielding the schemes ANMF-RSCM and 
ANMF-RTE. One major bottleneck toward determining the 
regularization parameter that optimizes the performances of 
the ANMF detector, is linked to the difficulty to clearly 
characterize the distribution of the ANMF statistics under the 
cases of presence or absence of a signal of interest (Hi and 
//(i). In order to deal with this issue, we considered the regime 
under which the number of samples and their dimensions 

6 Note that we do not compare with the zero-regularization case as in the 
first experiment, since, contrary to the Gaussian case, we do not have in our 
disposal theoretical results allowing the tuning of the threshold to the value 
that achieves the required false alarm probability. 


- Proposed design 

-Design using p 0 mia, f 0 mi a [ 15| 



Fig. 3. ROC curves of ANMF-RTE designs for a = 0.1, 0.25, 0.5, p = a(0) 
with 9 = 20°, N = 30, n = 60: K distributed clutter setting 


•s 

o 


a 

o 


<u 

Q 



Fig. 4. Detection probability with respect to a, p = a (9) with 9 = 20°, N = 
30, n = 60, v = 0.1,0.9,30: K-distributed clutter, false alarm probability 
=5%. 


grow large simultaneously. Based on tools from random matrix 
theory along with recent asymptotic results on the behaviour of 
the RTE, we derived the asymptotic distribution of the ANMF 
detector under hypothesis and H \. The obtained results 
have allowed us to propose an optimal design of the regu¬ 
larization parameter that maximizes the detection probability 
while keeping fixed the false alarm probability through an 
appropriate tuning of the threshold value. Simulations results 
clearly illustrated the gain of our method over previously 
proposed empirical settings of the regularization coefficient. 
One major advantage of our approach is that, contrary to first 
intuitions, it leads to simple closed-form expressions that can 
be easily implemented in practice. This is quite surprising 
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given that the handling of the classical regime where n grows 
to infinity with N fixed has been shown to be delicate. As 
a matter of fact, it has thus far been considered only for the 
non-regularized Tyler estimator where intricate expressions in 
the form of integrals were obtained HD- Building the bridge 
between both approaches is an open question that deserves 
investigation. 
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Appendix A 
Proof of Theorem[3] 


The proof of Theorem [3] consists of two steps. First, we 
study the asymptotic behaviour of the detection probability 
for fixed p. Then, by a similar argument to the one considered 
in l28l . we establish the uniformity of the result over the 
considered set of p. Assume that the received signal vector 
y is given by: 


y = 7w p+x 


with ||p|| 2 = N and let us write VNT^ scm ( p) as: 


VNT^ SCM {p) = VN- 


^yR^(p)-a 


Vn 


hy* Ti N 1 (p)y 


/ P*R J y 1 (p)p 


N 


A close inspection of the expression of VNT$ scm ( p) re¬ 
veals that the fluctuations will be governed by the numerator 
y*Rj v 1 (p)-^= since, from classical results of random matrix 
theory, we know that quantities in the denominator exhibit 
a deterministic behaviour, being well-approximated by some 
deterministic quantities. In effect, 

-^p*R^ 1 (p)p—-^p*Qjv(p)p-^4 0, (11) 

while: 

^y*R-\p)y-^-trC N Q N (p) ^ 0. (12) 

The first convergence ( II 11 ) follows from Theorem 1.1 of [HP 
whereas the second one is obtained by observing that, because 
of the low-SNR hypothesis: 


1 


1 


y* R-n 1 (p)y- A 7 X * V (p) x 


o 


N , -- N N - 

and then using the well-known convergence result 


-^-x*R Ar 1 (p)x— tr CnQn(p) -^4 0. 

We will now deal with the fluctuations of the numerator. We 
have: 


Vn 


»-i. 


N y ’ R « P 


^-R^p+x*R 


■tv 1 


P 

Vn 


Arguing in a similar way to that in ( II Ik we know that the 
quantity -£ap*Rjy L p does not fluctuate and converges to: 

-^ap*R^ 1 p--^-p*Q iV (p)p 0, 


while, from 
—^=5ft(x* Rj^p), 


Vn 


3(x*R jV l p) 


P*C JV Qir(p)p 


2p 2 N (l-cm(-p) 2 {l-p) 2 jj tvC 2 N Q? N (p)) 


Z =Op(l) 


for some Z 
Let r = 


■AT(0,I 2 ). 

1 R(x*R^p), 


—JV 

T v and u>n the quantities: 


1 3(x*R Ar 1 p) 


Vn 


. Denote by 


Tjv = 


1 


P*CjvQ^(p)p 


2p 2 N (l-cm(-p) 2 (l-p) 2 i trC^Q^(p)) 


wjv = -t^P*Qat(p)p 
N p 

Recall the following distance between probability distribu¬ 
tions: 


/3(P,P) 


= sup 


fdP- / /dip, ll/llsi < 1 


where ||/||bl = ||/||z,* P +||/lloo, ||/||l» p being the Lipschitz 
norm and ||.||oo, the supremum norm [?]. Assume for the mo¬ 
ment that lim sup Tat < oo and limsup j^tP*Qat(p)p < oo. 
The proof for these statements will be provided later. Then, 
from Theorem 11.7.1 in [?], 


£ ( r, -^ap*R 


'N P 


,£(i 


T N Z 


UJN 


-A 0 . 


where C(X) stands for the probability distribution of 
X. This in particular establishes that the random vari¬ 


able 


(r, -j^ap*R Jv 1 p 


tion to 


(t n Z ,ujn\ 
leorem in [?. 


ping Theorem in 

Vn £-y*R^ 1 p 

dom variable 


converges uniformly in distribu- 

From the uniform continuous map- 
Theorem 1], we thus prove that 
behaves asymptotically as a Rice ran- 
with location -^P*Qat(p)p and scale 


1 _ P*CjyQ^-(p)p _ 

2 P Z N (l-cm(-p) 2 (l-p) 2 -^ trC^Q^(p)) ’ 


Using this result 


along with Slutsky Lemma, we conclude that under Hi, 

T RSCM {p) 

is also asymptotically equivalent to a Rice ran- 


L N 

dom variate but with location 


\Zp*Qw(p)r 


VN ^itrC N Q,v(p) 

&n,scm- We therefore get, for a fixed p. 


Ti 


SCM 


N 


> -H-ffi 

Vn' 


-Q i 9scm{p), 


and scale 


0 


&n,SCM , 

The generalization of this result to uniform convergence across 
p £ 'IZ‘V' M can be derived along the same steps as in lf28l . We 
now provide details about the control of the lim sup T y and 
limsup wat. The fact that limsupcujv < oo follows directly 
from the last item in Assumption [U while the control of 
lim sup Tat < oo requires one to check that: 

lim inf 1 —cm(-p) 2 (l-p) 2 ^ trC^Q^(p). 
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The proof hinges on the observation that this term naturally 
appears when computing the derivative of m{z) with respect 
to z at z = —p. Simple calculations reveal that: 


i'(z) = z + - 


i l ~p) 

N 


tr CnQn(z) 


-2 


Proof: The proof relies on the Talor expansion of Hn and 
gisr in the vicinity of 1, which asserts that for any p £ [0,1] 
there exist and £2 satisfying: 

Mp) = h ' N (i)(p-i)+(p-i) 2 ^v(£i) 

9n(p) = g N {l)(p-l) + {p-lfg" N (&) 


x ( l—m(z) 2 (l—p) 2 — tr C%Q%(z) 


-1 


f p(dx) 1 _ 

(P l)^jv(£i)Siv(l-) (P 1)^tv(3)5iv(^2) 

/ ( x+p ) 2 re 2 

g'A 1 ) (Pn( 1 )+(p- 1 )5jv(6)) 


It suffices thus to show that m'(—p) is bounded. As m is a 
Stieltjes transform of some positive probability measure p, it 
can be written as: 


which ends the proof. 


Appendix B 

Proof of proposition^ 

For ease of notation, we denote by f(p) and f(p), the 
quantities /scm(p) and fscM^p )■ It is easy to see that f(p) 
and f(p) converges to an undetermined form as p f 1. Set 
f(p) = with g and h being given by: 

dip) = (1 ~P) {p*'R-n 1 (p)p) (1 ~P) (l-c+^PtrR^p)) 

Kp) = P*R-iv 1 (p)P-PP*R'iv 2 (p)P 

The handling of the values of p approaching 1 can be per¬ 
formed using the 1’HopitaTs rule. 

The idea of the proof is to treat seperately the values of p in 
the interval [re, 1 —re] and those in [1 —re, 1] for some re small 
enough. In order to allow for a setting of re that is independent 
from N, we need to prove that: 


We therefore have, 

hN{p) h' N (l) 


lim sup 

N 


9n(p) g' N (l) 


= lim sup 
N 


M 1 ) 


Pn( 1 ) + (p- 1 )5 , at(6) a ' N (!) 


limsupjy |fe jv (^i)g jv (l)|+limsup Af |fejy(£i)giv( 6 )l 
lim inf |. g [ v ( 1 )| 2 

Tending p to 1 establishes the desired result. ■ 

Obviously functions h(p) and g(p) satisfiy the assump¬ 
tions of Lemma 01 Applying l’Hopital’s rule and using the 

d -R —1 / 
dp^N 

^R^ 2 (p) = -2R^ 3 (p) Rat+I), we finally prove: 


<|P-1 


differentiation rules ^R w 1 (p) = —R N 2 (p) Rjv+I) and 


lim lim sup 

Pfl N 


kp)-- 


N 


= 0. 


P*RnP 

Now, using the fact ^-p*Rjvp— ^-p*CArp — 
tion to the last item in Assumption Q] we get: 


(14) 

0 in conjunc- 


lim lim sup 
Pt 1 N 


f(p)-- 


N 


0. 


(15) 


P*CtvP 

On the other hand, a careful analysis of the behaviour of f(p) 
near 1 reveals similarly that: 

N 


lim lim sup 

Pt 1 N 


f(p )- 


h N d) 


S/v(l) 


lim lim sup 

Pt 1 N 


f(p)~: 


= 0 . 


(13) 


P*C N p 

Combining (fl5] > with (fl6]> . we finally obtain: 


0 


(16) 


To this end, a uniform variant of the l’Hopital’s rule is 
essential. This variant is stated in the following Lemma: 

Lemma 13. Let /n{p) = ^(p) ^ N anc ^ 9 N being de¬ 

fined in the interval p £ [0,1], Assume that h]y(l) = giv(l) = 
0 while lim inf at > 0, lim sup at < +00 

and lim sup at 


lim lim sup 

pti AT 


f(p)~f(p) 


0 


p= 1 


p=i 


It then suffices to prove Proposition [4] on TZ K - [re, l — 

To this end, we need to recall the following relations satisfied 

by m N {-p): 

/ \ 1 —c cl . 

m N (-p) = - \--—trQ N (p) 


< +00. Assume also that the second 


pN 


p= 1 

derivatives of and gjy are uniformly bounded in N, that 
is: 

sup lim sup h N (p) < +00 
pe [0,1] n 


and 


m N {—p) = yp+c{l-p) — trC N Q N (p) 
Combining these relations, we therefore get: 


sup lim sup 
pe [0,1] n 


9n(p) 


< +00 


— tr CatQat(p) = 


p( 1 -^ tr Qiv(p)) 


Then, 


lim lim sup 

p-> 1 N 


hwip) h' N { 1) 


9n(p) g' N { 1) 


-> 0. 


N “ — (l-c)(l-p)(l-c+%ixQ N {p)) 

The result thus follows by using Proposition [2] and noticing. 


0, 


in the same way as in [[28], that: 

1 1 


sup 

PG[k,1—£] 


^trQAr--trR Ar 1 (p) 
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sup 

p(E [«,!—■£ 


and 


sup 
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S QNP ‘^F p,a » Wp 


4p*CjvQ^(p)p 


l-cm(-p) 2 (l-p) 2 ^ tr C 2 N Q, 2 N (p) 

b (p*R]^p-pp*R^p) 




-c4 tr Rjy * 1 11 (p 


'JV 


0. 
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